Computer hardware, firmware, and software are becoming ever-increasingly complex. As a consequence, so are the procedures, systems, and tests for testing them.
Commonly, testing organizations for various projects, even within the same company, work independently to develop and implement their own, different, testing methodologies and testing tools. Consequently, a variety of tools and test packages exist which meet specific testing needs but do not provide a general-purpose testing environment. This inhibits the sharing of tests among testing organizations. It also leads to much duplication of test development and test management work that has already been done by other organizations, because the prior work is typically embodied in a form incompatible with, and hence not useful to, other testing organizations.
Because other organizations are typically not easily able to build upon the work of others but rather start anew, existing testing methodologies generally lack the level of sophistication and refinement that would be expected of long-term on-going developments. For example, tests are typically not developed, executed, or maintained in a consistent manner. Managing and planning of the overall testing effort is difficult. Test development, execution, and validation is labor-intensive. Requirements and documentation are incomplete and do not conform to rigid standards. Information about tests must be obtained from the tests themselves, is inconsistent in type from project to project, and is difficult to extract. Test results are difficult to understand, and therefore to report. Attempts at test automation are rudimentary, and lack modularity and versatile interfaces such as would facilitate change and growth. And different projects use different--often vague to outsiders--non-standard testing terminology. Because inter-project test and test tool compatibility is so rare, test-sharing capability is typically not provided even among systems that may be compatible.
Hence, a problem in the art is lack of an arrangement that supports automation of the testing and test management processes and that enforces standardization of those processes, yet does so in a manner versatile enough to avoid imposing undesirable constraints on test archives of disparate testing organizations, that facilitates tool evolution and change, that provides users with information about tests and test results readily, easily, and in a consistent manner, that aids users in selecting and obtaining tests for execution, and that permits tests to be easily shared among a plurality of testing organizations.